In 1997 the Massachusetts group that insured state employees released hospital records with the names removed, so researchers could use them. A graduate student, Latanya Sweeney, bought the city's voter list for a few dollars. It held names, ZIP codes, birth dates and sexes. Only one voter in the governor's town matched the governor's birth date, and the record that matched him in the hospital file was his. Nothing in the hospital data said "Weld". It did not need to.
The same kind of story keeps repeating. A hospital program picks patients for extra care with a score that is quietly worse for one group. A benefits agency flags thousands of parents as fraudsters, and a government falls. The data was real, the maths was correct, and people were hurt anyway.
Data ethics is what a data scientist learns to see before the harm: privacy that dissolves when datasets meet, fairness that has more than one meaning, and decisions that nobody can appeal.
This piece describes how privacy law and data ethics work. It is not legal advice; laws differ by country, change often, and a real question belongs with a qualified lawyer or your local data-protection authority.
Removing names does not make data anonymous. What identifies you is the combination of ordinary details that others also know: where you live, when you were born, your sex. Each detail is shared by many people; the set may be shared by one. Sweeney later estimated that 87% of people in the United States were unique on just ZIP code, birth date and sex (2000, using 1990 census figures). A later re-analysis with 2000 census data (Golle, 2006) put it nearer 63%. Either way, three dull fields single out most people.
The lesson came back in 2008, when Arvind Narayanan and Vitaly Shmatikov showed that a handful of movie ratings and rough dates could pick one subscriber out of the Netflix Prize data.
Below, 24 invented patients. Blur the fields and watch the smallest crowd grow, and notice what you lose: a researcher who wants age-specific rates can no longer have them.
24 invented people and invented five-digit codes. Each square is a person; a red square is unique, and so identifiable by anyone who knows those fields. "Same condition" counts crowds in which everyone has the same diagnosis, which leaks it even when k is large.
Without JavaScript: with full ZIP, exact birth year and sex, all 24 people are unique. With the first three ZIP digits, a 20-year band and no sex field, the smallest crowd has 3 people.
k-anonymity has a known gap. If everyone in a crowd shares the same diagnosis, the crowd protects nothing: an attacker who knows you are in it learns your condition anyway. The deeper problem: the safety of a release depends on what else exists in the world, which you cannot see. Laws reflect this. The EU's GDPR treats data as personal if a person can be identified using "all the means reasonably likely to be used" (Recital 26), and it treats pseudonymised data, where names are swapped for codes, as still personal.
Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam Smith (2006) turned the question around. Instead of asking whether a table is anonymous, ask whether an answer would have been noticeably different if one person's data had been left out. If not, nobody can tell whether you were in the data, so nothing about you can be learned from the answer that could not be learned without you.
To get there, add random noise to the answer. For a counting query (how many patients have asthma?), one person changes the true count by at most 1. Adding noise from a Laplace distribution with scale 1/ε makes the two answers, with you and without you, almost indistinguishable. The number ε (epsilon) is the privacy budget: small ε means more noise and more privacy.
Invented clinic: the true count of asthma patients is 412 out of 1,000. The curves are exact Laplace densities, with and without one more person; the dots are random draws, so they change each time you ask.
Without JavaScript: at ε = 1 the noise has a typical size of about 1.4 and 95% of answers fall within about 3 of the truth; at ε = 0.1 about 95% fall within 30.
The cost is real and it is the whole point: a count of 412 plus or minus 3 is fine for a statistician, but the same method on a count of 12 plus or minus 30 is useless. Differential privacy works best on big groups and badly on small ones, which is exactly where individuals are most exposed. Every extra question spends budget, so a data holder must decide in advance how much total ε to give away. The US Census Bureau used differential privacy for its 2020 Census releases, and users argued over whether the noise damaged small-area counts. Apple (2016) and Google (RAPPOR, 2014) used a local version, in which noise is added on your own device.
The oldest local idea is Stanley Warner's randomised response (1965). Ask a sensitive yes/no question and let each person flip a coin first: heads, answer truthfully; tails, flip again and answer yes on heads, no on tails. Any single answer is deniable. Yet if 40% of the answers are yes, the true share is 2 × 0.40 − 0.5 = 30%. A "yes" is at most three times as likely from someone whose truth is yes as from someone whose truth is no, which is ε = ln 3, about 1.1.
Most countries now have a data-protection law. They share ideas (use data only for stated purposes, keep no more than you need, let people see and correct their records, secure it) but differ in scope, legal basis and enforcement. The explorer is a summary for understanding, not a guide to compliance.
Summaries, not advice. Details, thresholds and dates change; check the text of the law and the regulator's guidance. Fine figures are the headline maximums written into each law.
Without JavaScript: the EU's GDPR (applied from 25 May 2018) allows fines up to €20 million or 4% of worldwide annual turnover, whichever is higher. Brazil's LGPD (in force September 2020), China's PIPL (November 2021) and India's DPDP Act (2023, phased in from November 2025) are comparable in spirit.
Three differences are worth keeping. First, the basis: the GDPR lists six lawful bases (Article 6), of which consent is only one, so a company may process data without consent if, for example, it is necessary for a contract. California's law works by giving residents rights (to know, delete, opt out of sale or sharing) rather than requiring a basis for every use. Second, automated decisions: GDPR Article 22 gives people a right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects, with exceptions; how far that amounts to a "right to explanation" is argued by lawyers. Third, scope: the US has no single general federal privacy law, so it relies on sector laws (health, children's data, credit) and a patchwork of state laws. The EU's AI Act (Regulation 2024/1689, in force 1 August 2024, with duties phasing in from 2025) adds rules by risk level for AI systems, including limits on some uses and extra duties for "high-risk" ones.
Bias is usually in the setup, not the maths. A model predicts a label, and the label is rarely the thing we care about. We care about illness and measure cost; about danger and measure arrests; about job performance and measure past hiring.
In 2019 Ziad Obermeyer and colleagues studied a widely used US health-care algorithm that ranked patients for extra care. It predicted future health-care cost. But less is spent on Black patients than on white patients with the same illnesses, for reasons of access and trust. So, at the same risk score, Black patients were measurably sicker. The authors estimated that removing the disparity would raise the share of Black patients flagged for extra help from 17.7% to 46.5%. The algorithm was never told anyone's race. It did not need to be.
Invented illustration: 100 people in each group, with needs spread evenly from 1 to 100 in both. A program picks the 40 people (the top 20%) with the highest predicted cost. Group B's cost is lower by the gap at the same need.
Without JavaScript: with no gap, 20 of the 40 people chosen are in group B. At a 20% gap, 11 are; at a 40% gap, none are, although half of the 40 people with the greatest need are.
The same pattern, in other forms: face-analysis systems trained mostly on lighter faces do worse on darker ones (Joy Buolamwini and Timnit Gebru's Gender Shades, 2018, found error rates up to 34.7% for darker-skinned women against 0.8% for lighter-skinned men in three commercial gender classifiers); résumé screens learn from who was hired before. When data records past human decisions, the model inherits the decisions. Hence the habit graduates keep: ask what the label really measures, and who is missing or mismeasured in it.
A model that is used also reshapes the world it measures. Predictive policing is the textbook case. Send more patrols to the area with more recorded incidents, and you record more incidents there, because you are looking there. Kristian Lum and William Isaac (2016) showed with simulation how a system trained on police records could keep directing patrols to the same neighbourhoods; Danielle Ensign and colleagues (2018) proved that under some assumptions such a loop runs away.
The toy makes that visible. Two districts have the same true crime rate and a small head start in records (55 to 45). Each round, patrols are shared in proportion to records raised to a power γ, and records grow with patrols. At γ = 1 the split stays put. Above 1, a "hotspot" policy, the head start is amplified.
A stylised model after Ensign et al. (2018), computed in expected values. Real systems are messier, and many departments have changed or stopped using such tools. The point is the mechanism, not a forecast.
Without JavaScript: starting from 55 to 45, after 200 rounds district A holds 50% of the records at γ = 0, 55% at γ = 1, 89% at γ = 2 and 96% at γ = 3, though the true rates are equal.
"Make the model fair" sounds like one instruction. It is several, and a landmark result says you cannot always follow all of them. In 2016 ProPublica analysed the COMPAS recidivism score used in US courts and reported that among defendants who did not go on to be re-arrested, 44.9% of Black defendants had been labelled higher risk against 23.5% of white defendants. The maker, Northpointe, answered that the score was equally accurate for both groups: a given score meant about the same chance of re-arrest whatever the defendant's race. Both statements were true. They were different definitions of fair.
Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan (2016) and Alexandra Chouldechova (2017) showed that if two groups have different base rates, a score cannot satisfy calibration and equal error rates at once, except in the useless cases of perfect prediction or equal base rates. Chouldechova's identity makes it concrete: false-positive rate = base-rate odds × (1 − PPV) / PPV × true-positive rate, where PPV is the share of flagged people who truly have the outcome. If base rates differ and PPV and TPR are held equal, the false-positive rates must differ.
| Group A | Group B |
|---|
Invented scores for 1,000 people in each group, in ten score bins. In every bin the share who truly have the outcome equals the bin's score, so the score is perfectly calibrated for both groups. Group B has the higher base rate (57% against 43%).
Without JavaScript: with one cut-off at 0.6, flagged-but-innocent rates are 11.4% in group A and 23.6% in group B, while PPV is 78.3% and 79.3%. Setting A to 0.5 and B to 0.6 brings true-positive rates to 67.4% and 68.2%, but PPV becomes 72.5% against 79.3%.
This is a mathematical fact, not an engineering failure. It forces a choice about values: which mistake matters most, whom it falls on, and who decides. The field names no single right metric. A graduate insists that the choice is made in the open, and that the base-rate gap itself is examined: it often reflects unequal policing, access or history, which no threshold can fix.
Modern research ethics begins with harm done in the name of science. In the Tuskegee study (1932 to 1972) US Public Health Service researchers followed Black men with syphilis and withheld treatment after penicillin became standard, without informed consent. The Belmont Report (1979) answered with three principles that data ethics still borrows: respect for persons (consent and autonomy), beneficence (do no harm, maximise benefit) and justice (fair distribution of burdens and benefits). The Menlo Report (2012) adapted them for computing research.
Practical habits followed. Datasheets for Datasets (Gebru and colleagues, 2018) and Model Cards (Mitchell and colleagues, 2019) ask builders to write down what data a model was trained on, what it should not be used for and how it performs for different groups. The GDPR requires a data protection impact assessment (Article 35) before high-risk processing. And the Dutch childcare benefits affair shows what happens without appeal. Between 2005 and 2019 the Dutch tax authority wrongly accused roughly 26,000 parents of benefit fraud and demanded repayment of often tens of thousands of euros. The data protection authority later found its use of nationality unlawful and discriminatory, and the cabinet resigned on 15 January 2021. The failure was a system in which suspicion was automatic and correction nearly impossible.
Data ethics asks the practical version of the questions in Ethics (36/02): who is in this data, did they know, who is affected by the decision, and how can they challenge it?
Data about people is not neutral raw material. Combining it can identify them, labelling it can mismeasure them, and deploying a model can reshape the world it learned from. There are tools (noise, documentation, audits) and laws, and none of them removes the need to decide whose mistakes count most.
Try this: choose an app you use and find its privacy settings. Write down three kinds of data it holds about you and one way a stranger who combined them with a public list could pick you out. Then ask who gets to see the scores it computes about you.
Sources. Sweeney, "Simple demographics often identify people uniquely," Carnegie Mellon Data Privacy Working Paper 3 (2000); Golle, "Revisiting the uniqueness of simple demographics in the US population," WPES 2006; Narayanan and Shmatikov, "Robust de-anonymization of large sparse datasets," IEEE Symposium on Security and Privacy 2008; Dwork, McSherry, Nissim and Smith, "Calibrating noise to sensitivity in private data analysis," TCC 2006; Warner, "Randomized response," Journal of the American Statistical Association 1965; Obermeyer, Powers, Vogeli and Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations," Science 366 (2019); Buolamwini and Gebru, "Gender Shades," PMLR 81 (2018); Angwin, Larson, Mattu and Kirchner, "Machine Bias," ProPublica, 23 May 2016; Kleinberg, Mullainathan and Raghavan, "Inherent trade-offs in the fair determination of risk scores" (2016); Chouldechova, "Fair prediction with disparate impact," Big Data 2017; Lum and Isaac, "To predict and serve?" Significance 2016; Ensign et al., "Runaway feedback loops in predictive policing," PMLR 81 (2018); Gebru et al., "Datasheets for datasets" (2018); Mitchell et al., "Model cards for model reporting" (2019); Regulation (EU) 2016/679 (GDPR), Recital 26, Articles 5, 6, 22, 35 and 83; Regulation (EU) 2024/1689; The Belmont Report (1979); Dutch parliamentary and data protection authority reports on the childcare benefits affair. All outputs of the interactives were computed in node. Datasets in the interactives are invented. This piece describes law and research; it is not legal advice.